All committers have signed the CLA.

@shoeffner thanks for this. This PR is certainly easier to reason about (because of its size) than some of the others I've seen.

That said, the test failure on Linux shows that this needs some rethinking, as that flag is not available in the version we test on, fetched via:

sudo apt-get install -y pandoc

I assume this flag was added recently? I'm not sure how to handle this.

The Linux build seems to use pandoc 2.5, but --citeproc is only supported > 2.11 (https://pandoc.org/releases.html#pandoc-2.11-2020-10-11 – on macOS, 2.18 is installed, so I didn't notice) – before --filter pandoc-citeproc was the way to go.

If I remember correctly, pandoc-citeproc had to be installed manually, but I will check it out in a container and see what I can do!

choco and brew seem to install 2.18, so Windows and macOS should not need any specific handling.

On Linux we have everything from 2.5 (Ubuntu 20.04, e.g. GH actions) to 2.17 (Arch Linux); the most common version seems to be 2.9 (Debian stable, Ubuntu 22.04, and other distros), here pandoc-citeproc must be installed additionally.

This makes the pull request much more complicated than what I was aiming for, so feel free to close it again. My current approach is now:

if pandoc >= 2.11
    add --citeproc
else if pandoc-citeproc available
    add --filter pandoc-citeproc
otherwise
    keep the old behavior

If the citations are not supported (due to the version or pandoc-citeproc dependency), SupportsCitations is false and the corresponding tests are skipped. I also added a remark to the docs.
However, I was unable to test all configurations yet (i.e., Linux with newer pandoc, older pandoc, with and without pandoc-citeproc; nor did I test windows), thus I update the PR to "WIP".

An alternative could be to check for the version and ignore pandoc-citeproc. This would make it much simpler and whoever wants to use citeproc might also want to use other pandoc features anyways. But it puts some burden on the users who have to install a newer pandoc version.

What do you think?

I rebased the commits and squashed them.
Unless there is anything else or the CI fails again, the PR is for the moment done, so I will also remove the WIP label.

I updated the PR again, I made a mistake (err != nil instead of err == nil)which caused the tests to skip and never activated pandoc. It should now work as expected.

Rebased on master.

Sorry for the late reaction. I fixed it to use argsv := []any{"--version"} instead of collections.StringSliceToInterfaceSlice....

Rebased on master.

Okay, comparing strings 2.5 and 2.11 was a stupid idea... Now it has a proper type and a function to compare two instances.

3c5e31f#diff-f662bf93836a7230b77cdb532095a04622eb9eda3054124e8ba9399786870efaR88-R94

Additionally, I included some test cases for those comparisons. I hope it will work now ;-)

Unsupported pandoc should now cause the tests to SKIP instead of causing a panic.

This PR has been automatically marked as stale because it has not had recent activity. The resources of the Hugo team are limited, and so we are asking for your help.
Please check https://github.com/gohugoio/hugo/blob/master/CONTRIBUTING.md#code-contribution and verify that this code contribution fits with the description. If yes, tell is in a comment.
This PR will automatically be closed in the near future if no further activity occurs. Thank you for all your contributions.

Is this still relevant to Hugo? I have to admit that I don't even use it right now, so I haven't followed up anymore.
But if it's still useful, I am happy to rebase/update/work on the PR – otherwise, feel free to close it :)

Is this still relevant to Hugo? I have to admit that I don't even use it right now, so I haven't followed up anymore. But if it's still useful, I am happy to rebase/update/work on the PR – otherwise, feel free to close it :)

I still haven't figured out a good way to make this work in a different way. For me, just having hugo pass --citeproc would be a game-changer; reading csl:, bibliography: form the front-matter and enabling link-citations by default would be the whole package. For now, I have to do a lot of manual work for any content with references.

So I would love for this PR to be rebased!

The docs moved around quite a bit over the last 1.5 years, so I moved the section about citations to its own markdown file during the rebase. Please let me know if I should change that (or, for example, the weight) – or simply change it.

I also found out that the test setup (or provider config) seems to have changed, I am currently working on updating that.

I found the issue in the test setup (the ProviderConfig was not passed correctly, and I seem to have written that bug the last time I rebased in August '23, whoops).

There's also been some output format changes in Pandoc, so I will relax the tests a bit and make sure they only check for "contains" on the references, not the exact markup (as that will be different between Pandoc versions).

Done, the tests no longer rely on a specific output format and only check if the author last names, the year, and a word from the title are included in the documents – or not included if no citations was there.

I received an email that the tests have failed, but I think that's what I already fixed in the latest commit.

Just tested this rebase and it works fine; I built with $ CGO_ENABLED=1 go build -tags extended . after extracting.

My current workflow with regular extended Hugo:

• Unprocessed markdown content goes in ${DRAFT_DIR}
• CI pipeline preprocesses with pandoc and some ugly kludges:

touch .dummy;
ls -1 ${DRAFT_DIR}/*.md \
| xargs basename \
| xargs -i pandoc-${PANDOC_VERSION}/bin/pandoc \
-C -t gfm --resource-path=${DRAFT_DIR} -H .dummy \
${DRAFT_DIR}/{} -o ${POST_DIR}/{}

• Build site with hugo

Workflow with this PR:

• Markdown content lives in content/
• Add markup: pandoc to front matter
• Whitelist pandoc executable in hugo.yaml
• Build site with hugo

Much easier and no preprocessing needed!

@shoeffner I'm not in a position to judge the value of this addition, but I do have a question. Instead of all of the version checking code, can you just test for success?

pandoc --citeproc --dump-args
echo $? # 0 (success)

pandoc --foo --dump-args
echo $? # 6 (failure)

Add markup: pandoc to front matter

To clarify, this is because I did not set pandoc as the global markup engine, only for select content.

To clarify, this is because I did not set pandoc as the global markup engine, only for select content.

Why not change the file extension to .pdc or .pandoc?

Why not change the file extension to .pdc or .pandoc?

Because I didn't know I could do that. Thanks for the tip - I'm not a hugo expert by any means!

Workflow with this PR:

• Markdown content lives in content/
• Add markup: pandoc to front matter
• Whitelist pandoc executable in hugo.yaml
• Build site with hugo

Much easier and no preprocessing needed!

It is not yet available on hugo 0.145.0.

I changed the citeproc detection to use --dump-args, as suggested by @jmooring . It seems to work for me, so feel free to test it. I also rebased on the latest master version again.

Hm, it seems the tests were unable to pull some data from X.com - I don't think that's related to my PR, though. Or am I mistaken?
The tests I wrote seem to have worked (markup/pandoc/convert_test.go).

Hm, it seems the tests were unable to pull some data from X.com - I don't think that's related to my PR, though. Or am I mistaken?

The X.com API has been unstable lately.

You' re logging and returning errors, resulting in repetitive console output:

Start building sites … 
hugo v0.146.0-DEV-038007c117257f13acde3d2d230003bfd8161b9b+extended+withdeploy linux/amd64 BuildDate=2025-03-18T17:16:55Z VendorInfo=jmooring

ERROR Could not call pandoc: access denied: "pandoc" is not whitelisted in policy "security.exec.allow"; the current security configuration is:

[security]
  enableInlineShortcodes = false

  [security.exec]
    allow = ['^((dart-)?sass|git|go|npx|postcss|tailwindcss|gnome-text-editor|nano|asciidoctor|babel|xpandoc|rst2html)$']
    osEnv = ['(?i)^((HTTPS?|NO)_PROXY|PATH(EXT)?|APPDATA|TE?MP|TERM|GO\w+|(XDG_CONFIG_)?HOME|USERPROFILE|SSH_AUTH_SOCK|DISPLAY|LANG|SYSTEMDRIVE)$']

  [security.funcs]
    getenv = ['^HUGO_', '^CI$']

  [security.http]
    methods = ['(?i)GET|POST']
    urls = ['.*']
Total in 14 ms
Error: error building site: render: failed to render pages: render of "/home/jmooring/code/hugo-testing/content/_index.pdc" failed: "/home/jmooring/code/hugo-testing/layouts/_default/home.html:11:5": execute of template failed: template: _default/home.html:11:5: executing "main" at <.Content>: error calling Content: "/home/jmooring/code/hugo-testing/content/_index.pdc:1:1": access denied: "pandoc" is not whitelisted in policy "security.exec.allow"; the current security configuration is:

[security]
  enableInlineShortcodes = false

  [security.exec]
    allow = ['^((dart-)?sass|git|go|npx|postcss|tailwindcss|gnome-text-editor|nano|asciidoctor|babel|xpandoc|rst2html)$']
    osEnv = ['(?i)^((HTTPS?|NO)_PROXY|PATH(EXT)?|APPDATA|TE?MP|TERM|GO\w+|(XDG_CONFIG_)?HOME|USERPROFILE|SSH_AUTH_SOCK|DISPLAY|LANG|SYSTEMDRIVE)$']

  [security.funcs]
    getenv = ['^HUGO_', '^CI$']

  [security.http]
    methods = ['(?i)GET|POST']
    urls = ['.*']

I removed the two log lines, is it better this way? Otherwise, can you share the example you are trying so that I can reproduce the behavior?

@shoeffner

When you ask a question such as, "Is it better this way?" it sounds like you are guessing, which doesn't inspire much confidence. The example I ran is indicated by the error message: I removed pandoc from the allow list.

To answer your question, yes, it is better this way.

@jmooring, I only ever run the tests with mage -v check and they do not produce any meaningful output as they suppress it all.

I don't know what example project you use to test the PR, even though it was clear to me that you removed pandoc from the allow list.
So please excuse that I asked whether it was better; what I wanted to ask is whether this was closer to your preferred behavior, as I was trying to incorporate your feedback. I did not mean to offend and I am sorry if I did.

Of course, I understand that I am not the ideal contributor to work on this because I haven't worked with Hugo since shortly after I originally drafted this PR in 2022, as such, I am not even using this feature myself anymore. I am trying my best to get a feel for the project and improve as I get suggestions, be it by you, @bep - who reviewed the first draft and helped me over a few Go hurdles -, or others in this thread.

If you feel I am not confident enough on contributing this, I will of course step back from the PR, you can close it, and/or someone else can pick it up.